# Lightweight inference

A quantized version based on the Baidu ERNIE-4.5-0.3B-PT model, optimized through the llama.cpp tool to reduce the model size and improve the running efficiency.

This project provides the GGUF quantized file for the Echelon-AI/Med-Qwen2-7B model, supported by Featherless AI, aiming to enhance model performance and reduce operating costs.

This is a 3-bit quantized version converted from the mistralai/Devstral-Small-2505 model, designed for the MLX framework and supports multilingual text generation tasks.

Qwen3 is the latest version of the Tongyi Qianwen series of large language models, offering a range of dense and Mixture of Experts (MoE) models. Based on large-scale training, Qwen3 has achieved breakthrough progress in reasoning capabilities, instruction following, agent functionalities, and multilingual support.

Kimi-VL-A3B-Thinking-6bit is a multilingual vision-language model converted based on the MLX format, supporting image-text to text tasks.

This is a 3B-parameter language model optimized for Korean, converted to GGUF format for compatibility with llama.cpp.

Llama-4-Scout-17B-16E-Instruct-GGUF is a quantized version based on the Llama-4-Scout-17B-16E-Instruct model, supporting multilingual processing and suitable for chat and instruction tasks.

Image depth estimation model based on Latent Bridging Matching (LBM) technology, achieving rapid image transformation through latent space bridging

A Persian-specific model fine-tuned based on the Gemma 3 architecture, utilizing QLoRA 4-bit quantization technology, suitable for running on ordinary hardware.

This is a GGUF format quantized version of the mrfakename/mistral-small-3.1-24b-instruct-2503-hf model, suitable for text generation tasks.

This is a GGUF format model converted from Google's Gemma 3.4B parameter model, suitable for text generation tasks.

Gemma 3.4B IT is a lightweight open-source large language model released by Google. Based on a parameter scale of 3.4B, it is suitable for dialogue and instruction following tasks.

Phi-4-multimodal-instruct is a lightweight open-source multimodal foundation model that integrates language, vision, and speech research data from Phi-3.5 and 4.0 models. It supports text, image, and audio inputs to generate text outputs, with a context length of 128K tokens.

GGUF format model converted from AtlaAI/Selene-1-Mini-Llama-3.1-8B, suitable for text generation tasks and supports multiple European languages.

This model is converted from deepvk/USER-bge-m3 into GGUF format, primarily used for sentence similarity and feature extraction tasks.

This model is a GGUF format version converted from Google's flan-t5-base model, supporting multiple languages and tasks, suitable for text generation and reasoning tasks.

An instruction fine-tuned model based on Gemma 2 Baku 2B, which optimizes the instruction following ability and is suitable for natural language processing tasks.

This is Meta's 1 billion parameter instruction-tuned model from the Llama 3.2 series, converted to GGUF format for use with llama.cpp

A large language model developed by the National Institute of Informatics in Japan, supporting Japanese and English, with instruction fine-tuning capabilities.

This model is converted from deepvk/USER-bge-m3 to GGUF format, primarily used for sentence similarity calculation and feature extraction.

ONNX version of the Dutch Forensic Institute's RobBERT-2022 Dutch sentence embedding model, optimized for high speed and lightweight performance.

This is the CTranslate2 converted version of OpenAI Whisper large-v2 model for efficient speech recognition

A model based on sentence-transformers for determining the relevance between short texts and questions.

An ultra-small model trained on a bean dataset, primarily for testing and demonstration purposes.